r? @Aatch

(rust_highfive has picked a reviewer for you, use r? to override)

Nice! This feels like it's a bit ad-hoc though and it'd unfortunately be an insta-stable interface to the compiler. For something like cargo check I don't think this is necessary as 99% of the time all your dependencies are already built, but I've wanted this for other aspects such as documenting other platforms and/or pipelining compilations.

Some specific concerns I have are:

• This just uses the same rlib format, but the object file is notably missing. This can lead to surprising errors if you try to actually link against those rlibs. I imagined this would create a whole new separate file type (e.g. *.rmeta) which the compiler would then give a first-class error message about not being able to link against if that is requested.
• The relationship between -Z no-trans and --emit=rmeta is somewhat odd to me, I would expect the compiler to naturally stop at the right point if only --emit=rmeta is present (e.g. as it does with other output types today).
• Implementation-wise I'd prefer to see rmeta have a more first-class position instead of being a few targeted if-else conditionals on the normal rlib path as these two forms may diverge over time.

r? @alexcrichton I've got no opinion on the design issues.

... So GitHub/chromium managed to lose my comment, and I'm not retyping all that.
The important thing is that I don't want this merged in its current state, I've barely begun understanding the code involved here.
But I do want @bstrie to asses whether this would work for cargo check.

The reason -Z no-trans isn't "implied" by --emit=rlibmeta is because the metadata actually changes after trans, as it gets symbol names and a few other things (that only influence trans in downstream crates).

Should this kind of metadata always be emitted before trans, with certain information missing?
Maybe the extra information could be split out into a different bit of metadata that is emitted after trans?

That would make sense if it was going to be used to pipeline compilations.

But I do want @bstrie to asses whether this would work for cargo check.

With rust-lang/cargo#1701, I believe that cargo check is basically cargo rustc -- -Z no-trans, so I'd recommend starting to investigate there (before taking this route)

The reason -Z no-trans isn't "implied" by --emit=rlibmeta is because the metadata actually changes after trans, as it gets symbol names and a few other things (that only influence trans in downstream crates).

Interesting! I had forgotten about this. This poses an interesting problem that may be difficult to overcome, unfortunately. In terms of pipelining compilations, there are two synchronization points which need to happen if we have two sets of metadata:

1. Once the first phase of metadata is produced, immediate downstream dependents can start compiling.
2. Once I'm ready to trans, all upstream dependencies need to have produced the second phase of metadata before I can continue.

I feel like managing multiple synchronization points in the compiler could get quite complicated (even one is nontrivial), and if we take a route like this I'd much prefer to minimize the complexity as much as possible.

In the ideal world we could do something like have a "dry run" of trans which doesn't actually translate anything (but calculates symbol names), or perhaps even just have the ability to calculate symbol names ahead of trans. That way there'd only need to be one metadata format that's the same across dylibs, rlibs, and standalone files.

Ah, that's a good point, I had forgotten how deterministic everything should be - maybe it can be computed ahead of trans (and... cached, so trans is more efficient at it. may also be possible to run trans from the metadata alone).

I would also be somewhat more comfortable landing this ahead of time if the ability to emit a metadata file is tied to the ability to use unstable features (e.g. this is only enabled on nightly).

Ideally I'd also like cargo check to be a precursor to allowing dependents to begin compiling while their dependencies are still being codegenned, which to me implies that we'd want separate files for metadata and linkable artifacts. Alex, in the past when I've asked you about this you seemed to agree that separate files was a good idea, and that we might want to consider making separate files the normal mode of compilation regardless (though I could be misremembering).

Bikeshed: Is rlibmeta somehow specific to rlibs? Don’t we bundle metadata with shared libraries as well? If so, could this be named just --emit=metadata instead?

Alex, in the past when I've asked you about this you seemed to agree that separate files was a good idea, and that we might want to consider making separate files the normal mode of compilation regardless (though I could be misremembering).

Yeah I definitely think we'll want a separate and dedicated file for this, but I don't think we should be producing one in all cases. It's quite nice that an rlib and/or dylib is a standalone artifact that doesn't need to be paired with something else, so I'd prefer to keep it like that if possible.

Don’t we bundle metadata with shared libraries as well? If so, could this be named just --emit=metadata instead?

Good point! I'd also be fine with --emit=rmeta (rust metadata)

@nagisa shared libraries use compressed metadata - there was also "rlib" in there because, well, it's a "valid" rlib right now (if you don't try linking with it).
Definitely not keeping "rlibmeta" or the whole archive around the metadata.

☔ The latest upstream changes (presumably #26351) made this pull request unmergeable. Please resolve the merge conflicts.

Closing out of inactivity now, but I'm excited to see where this goes in the future!

I just realized that crater might have an use for this. Could be a good reason to revive the branch.